node:crypto: take the default RSA padding for a falsy padding option - #45359

Repro. Each call below throws on Bun 1.4.3 and returns on Node v26.3.0. With this PR each returns the same result as the call without padding.

import crypto from "node:crypto";
const { publicKey, privateKey } = crypto.generateKeyPairSync("rsa", { modulusLength: 2048 });
const message = Buffer.from("the same message");
const signed = crypto.privateEncrypt(privateKey, message);
const sealed = crypto.publicEncrypt(publicKey, message);
for (const padding of [null, 0, -0, false, NaN, "", 0n]) {
  crypto.privateEncrypt({ key: privateKey, padding }, message); // same bytes as `signed`
  crypto.publicDecrypt({ key: publicKey, padding }, signed); // `message`
  crypto.privateDecrypt(privateKey, crypto.publicEncrypt({ key: publicKey, padding }, message)); // `message`
  crypto.privateDecrypt({ key: privateKey, padding }, sealed); // `message`
}

Input space. A probe ran 27 padding values through the four calls on Node v26.3.0 and on this branch. Falsy values (undefined, null, 0, -0, false, NaN, "", 0n) give the default on both. Truthy values go through the integer conversion on both: true, 1 and "1" are RSA_PKCS1_PADDING, 4, 4.7, [4] and 2 ** 32 + 4 are RSA_PKCS1_OAEP_PADDING, and "0", " ", "abc", {}, [], new Number(0), -1 and Infinity are rejected as a padding mode. 1n and Symbol() throw a TypeError on both.

Two differences from Node remain. Both exist on main and come from BoringSSL, not from this read:

• The code names of the library errors differ, for example ERR_OSSL_EVP_ILLEGAL_OR_UNSUPPORTED_PADDING_MODE on Bun and ERR_OSSL_ILLEGAL_OR_UNSUPPORTED_PADDING_MODE on Node with OpenSSL 3.
• privateDecrypt with RSA_PKCS1_PADDING throws ERR_INVALID_ARG_VALUE on Bun. BoringSSL has no implicit rejection, and Node's PublicKeyCipher::Cipher has the same branch for that case.

Tests. The seven falsy cases fail on Bun 1.4.3 and pass on the debug build of this branch. The three truthy cases ("0", {}, []) pass on both. They hold the other side of the rule: only a falsy value means the default, so a fix written as "integer 0 means the default" would fail them.

Suites run on the debug ASAN build of this branch, all pass:

• test/js/node/crypto/crypto-rsa.test.js, also with BUN_JSC_validateExceptionChecks=1
• test/js/node/crypto/node-crypto.test.js
• test/js/node/crypto/crypto.key-objects.test.ts
• test/js/node/test/parallel/test-crypto-rsa-dsa.js
• test/js/node/test/parallel/test-crypto-sign-verify.js (it asserts that createVerify().verify({ key, padding: null }) throws ERR_INVALID_ARG_VALUE)
• test/js/node/test/parallel/test-crypto-private-decrypt-gh32240.js
• test/js/node/test/parallel/test-crypto-publicDecrypt-fails-first-time.js
• test/js/node/test/parallel/test-crypto-oaep-zero-length.js
• test/js/node/test/parallel/test-crypto-key-objects.js

Siblings. Node's lib/internal/crypto has two other reads where a falsy value means the default: the generator of createDiffieHellman (diffiehellman.js:115) and the point format of ECDH (diffiehellman.js:272). For undefined, null, 0, false, NaN and "", both give the same result on Bun 1.4.3 and on Node v26.3.0.

History. The check for undefined only dates from the native port of these calls in #19040.

How the numbers were measured. JSCipher.cpp was compiled two times, from main and from this branch, with the flags of the release build minus the LTO flags. nm -S gives the size of Bun::rsaFunction: 0x116a and 0x1232. The instruction counts come from the disassembly of the two objects, on the path from the load of the padding value to the next statement. With no padding on the key object the read costs 13 instructions, up from 3. With an integer padding it stays at 12. toBoolean is inline and calls nothing for a number, undefined, null or a boolean. The change adds no allocation and no call.

Not in this PR. rsaFunction reads padding, oaepHash and oaepLabel before it prepares the key. Node prepares the key first. The result is a different error on input that is invalid in two ways, for example publicEncrypt({ key: "not a pem", oaepHash: "nope" }, buf). This is a separate change. #45335 rewrites the key preparation in the same function.